Entropy Bounds and Statistical Tests

We convert a generic class of entropy tests from pass/fail to a measure of entropy. The conversion enables one to specify a fundamental design criterion: state the number of outputs from a noise source required to satisfy a security threshold. We define new entropy measurements based on a three-step strategy: 1) compute a statistic on raw output of a noise source, 2) define a set of probability distributions based on the result, and 3) minimize the entropy over the set. We present an efficient algorithm for solving the minimization problem for a select class of statistics, denoted as “entropic” statistics; we include several detailed examples of entropic statistics. Advantages of entropic statistics over previous entropy tests include the ability to rigorously bound the entropy estimate, a reduced data requirement, and partial entropy credit for sources lacking full entropy. 1. Good Entropy Sources and Tests Consider a four-sided die (with faces labeled 1,2,3,4) that is weighted so that the probability of the value i’s being rolled is pi. If each pi has the value of 1 4 , then we consider the die a fair die; we have no problem using the fair die as a source of randomness with each roll producing two bits of randomness. If the values of pi are not equal, then the die is considered weighted. There are a number of questions about the weighted die’s output that arise. Question 1.1. Can the weighted die be used as a good source of randomness? Question 1.2. How many rolls of the die are required to generate n random bits? Question 1.3. How do the answers to the previous questions differ if the values of pi are unknown versus if the values of pi are known? Question 1.4. What complications arise if one considers a die with a large number of sides versus a die with four sides? The concept that a source can lack full entropy and nevertheless be an excellent source of randomness has eluded some designers of random sources. There are suites of statistical tests that test for full entropy in a source: the source fails the test if the probability is not uniformly distributed among the possible output states. In particular, the weighted die would fail the statisical tests. One goal of this paper is to convert the pass/fail tests into a measure of entropy: the weighted die would receive partial credit as a source of randomness. With 1 the proper understanding of the entropy produced by the source one can answer Question 1.2, however the actual generation process is beyond the scope of the paper. The simple approach to Question 1.2 is to generate enough output to model the pi then compute the entropy of the modeled distribution. Three complications that often arise with this approach: the amount of output may be limited, outputs may be dependent, and there are many notions of entropy. In section 2, we discuss the notions of entropy. When data is limited, it may be impossible to estimate a probabilty distribution for the data; however, one may be able to compute a derived value that is used to estimate the entropy of the outputs. Most of this paper addresses issues related to this computation including: 1. What is the best statistic to compute from a random source? 2. How does one go from a statistic to a measure of randomness? 3. How does one combine multiple statistics to refine an estimate of entropy? The most difficult complication to address is dependency in outputs—we attempt to address this difficulty by admitting a simple dependent model of the data and performing analysis based on this model. It may be impractical or impossible to obtain an accurate model of the dependence between outputs; thus, we limit our scope of dependence to Markov processes. 2. Rényi Entropy There are many measures of disorder, or entropy. In [1], Rényi developed a class of entropy functions parameterized by a value α ∈ [1,∞]: (2.1) Hα(p) = 1 1− α log2 ( n ∑